Based on the peer-to-peer (P2P) technology, a distributed storage system can make use of all storage capacity of connected peers, i.e. nodes, to store data, such as duplicating important data in different peers, or splitting large-size data and sharing or storing them in different peers.
The present document relates to the specific Material Exchange Format (MXF). MXF is an open file format targeted at the interchange of audio-visual material with associated data and metadata. It has been designed and implemented with the aim of improving file based interoperability between servers, workstations and other content creating devices.
Basic MXF Structure
As shown in FIG. 1 taken form the MXF standard specification, the MXF Format consists of a File Header, a File Body and a File Footer.
The File Header contains Header Metadata, which provides information about the file as a whole, including labels for the early determination of decoder compliance.
The File Body comprises picture, sound and data essence stored in Essence Containers. Essence Containers from different tracks may be interleaved or separated as shown in FIG. 2. Body Partition Packs of File Body may optionally contain repetition of Header Metadata.
The File Footer terminates the file and may optionally contain Header Metadata. The File Footer may further include some information not available at the time of writing the File Header (such as the duration of the file). In certain specialized cases, the File Footer may be omitted.
The File Header is designed to be small enough that it can easily be isolated and sent to a microprocessor for parsing. The bulk of the file will usually be the File Body—this is the picture, sound and data essence. The File Footer provides capacity to put the Header Metadata at the end of the file. The reason is: in certain applications, such as recording a stream to an MXF file, there will be Header Metadata values that will not be known until the recording is finished. The File Footer provides a mechanism for doing this. It also provides clear indication that the file has terminated.
The basic MXF structure supports embedded metadata throughout the MXF file.
MXF Interoperation with Stream Interfaces
MXF files will often be processed in streaming environments. This will include streaming to and from videotapes and data tapes, and transmission over unidirectional links or links with a narrow-band return-channel.
Sequential writing is necessary when the source or the link or the destination operates only in streaming mode. Random access writing is permissible before or after data transfer, for example, to optimize downstream access performance.
MXF's Operational Patterns have a special qualifier bit that indicates that the file has been created for streaming.
MXF files may be directly created from standardized formats such as MPEG2 systems and elementary streams, AES3 data streams (Audio Engineering Society Standard) and Digital Video (DV) DIF packet streams. These formats may be mapped from one of several real-time interfaces such as SMPTE 259M (Serial Digital Interface, SDI), SMPTE 305M (Serial Digital Transport Interface, SDTI), SMPTE 292M (High-Definition Serial Digital Interface, HD-SDI), or transport interfaces with real-time protocols such as IEEE-1394, ATM, IEEE802 (ethernet), ANSI Fibre Channel and so on.
When a streaming file is captured, a File Header is created and the essence is KLV (Key Length Value) wrapped on the fly. The data rate increases due to the KLV wrapping and addition of headers. Real time streaming devices must ensure that any buffering requirements of a streaming interface are catered for with this change of data rate.
The basic MXF structure supports stream recording and playback.
File Splitting Requirement of a Distributed Storage System
The behaviour of a distributed storage home network system will in the future probably be characterized by a convenient automatically “storekeeping” of all kind of data. The user will not know in any case where the content is stored. So this kind of distributed storage system can be seen as a monolithic storage block and the system has to look by its own for a storage device that matches capacity of free storage without any administrative effort by the user, and with a transparent behaviour for the requiring application.
The distributed storage system has to deal with different lengths of application packets coming along with the streaming data of different standards (e.g. MPEG-2, DV, etc.). When streaming data has to be handled by the distributed storage system, it is not generally possible to get the size of the data in advance. So seamless real-time splitting of data could be necessary when the maximum capacity of a peer, or node, will be reached. This invention proposes methods for the processing of streaming data in the distributed storage environment for the record and playback condition.